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ABSTRACT 

The semiotic interpretative context is extended to 
program evaluation. Assumptions about the semiotic process of 
constructing, describing, and judging reality are examined in terms 
of the viability of the process in explicating a perception-based 
evaluation model. Models from which inspiration was drawn include R. 
W. Tyler's instructional objectives model; M. Provus' discrepancy 
model; the Context, Input, Process^ and Product model; M. Scriven's 
Goal-Free model; and R. E. Stake's model of evaluation. Five core 
evaluation signs are analyzed in terms of u recently completed case 
study of perception-based evaluation of a cross-national cultural 
awareness training program funded by an international foundation. 
Finally, a semiotic analysis for evaluation studies is discussed in 
terms of 13 Likert-scaled polemics. It is concluded that evaluation 
should be approached as both a native undertaking and an alien 
undertaking, and from an insider viewpoint and a disciplined outsider 
perspective. What is needed is a paradigm that negotiates major 
questions openly with all participants, gathers quantitative and 
qualitative scores of evidence, and applies standards or value 
criteria that are neither elitist from the perspective of the 
evaluator nor captured by the tacit agendas of the stakeholders. Two 
tables and a 72~item list of references are inclv^'ed. (TJH) 
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Background 

According to the thesis articulated by Graubaid (1972), the task for educational reform and 
social change is not to initiate further studies but to equip individuals with the means to understand 
and struggle against structural and attitudinal impediments to implementing known solutions. Only 
an integrated and balanced division of effort along diverse psychological, social, political and 
econoKiic fronts can enhance the value and utility of education and focus on basic priority needs of 
people. But the challenge still remains-how to transcend the rhetoric of "integrative" or 
interpretive approaches into action. 

Sebeok (1986) offers semiotics as a context for reconceptualizing foundations in the human 
sciences given its ability to transcend dichotomous and reductionistic methods of perceiving with a 
worid view that is holistic, respects complexity, and fosters synthesis. What the semiotic 
perspective can mean for educational researchers was recendy analyzed by Driscoll and Hynn 
(1989). Specifically, they identified five assumptions of a research semiotic: 

(1) systems are open aid life is disorderly; 

(2) behavior is modulated by immediate environment, species history, and individual 
history; 

(3) the eventual effects of signs is what constitutes meaning, for the individual, a 
community and ultimately what is accepted as "truth". In feet, Cunnin^am (1989) 
advocates "the arbitrary nature of these signs is not readily apparent to the luman organism 
until they are exposed to cultoral systems which depart ftom their own"; 

(4) unities are assumed, not dichotomies (e.g., objectiv^subjective, 
qualitative/quantitative, and individual/system; 

(5) reality is considered as the "final setfled opinion of an indefinite community of 
scientific investigators," because we humans "can never get closer to that ultimate reality 
than our perception and reasoning powers can take us." 

Purpose 

The purpose of this paper is to extend this interpretative context to program evaluation. 
Assumptions about the semiotic process of constructing, describing, and judging reality (i.e., our 
Umwelt) will be examined in terms of its viability in explicating a perception-based evaluation 
model (Kunkei & Tucker, 1983). Five core "signs g., the interpretive process inherent in major 
questions of focus, sources of evidence and standards of quality) will be analyzed in terms of a 
recently completed case study of a perception-based evaluation of a cross-national cultural 
awaicness training program funded by an international foundation. And finally, a semiotic 
analysis for evaluation studies will be discussed in terms of 13 Likert scaled polemics. 

Even vwtii extensive developments in the fields of research and evaluation, most decisions 
about educational programs continue to be made without systematic evaluation. Annually, 
decisions about budgets, current programming, and future planning are often determined by 
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political expediency. Often these decisions are based on unfounded opinions and insufficient 
infonnation. What occurs is a form of evaluation, but evaluation without a tiieoretical base, widiout 
systematic methodology, and where values and judgments are egocentric, elusive, and/or covert. 

Since the late 1960's, autiiors in program evaluation have addressed the need for systematic 
models. Undoubtedly, the use of models to order educational processes facilitates the conception 
of problems and the perception of interrelationships vwthin these problem contexts. While each 
model may view the same set of interrelationships, it is inevitable that certain concerns of the model 
builders will differ" due to personal frames of reference (AUdn, 1968). Dillon (1981) extends tiiis 
personal dimension of modelling to include "prefebricated configurati.ons of concepts" or ' items 
of coherence that we expect to find". 

The modeling of social systems involves another potential danger. Educaaon's use of 
analog models drawn from the "hard" sciences is unsound to the extent that empirical models 
represent only "observable" realities. The human dimension in education requires a model which 
represents both observable and subjective realities. And the dimension of change inherent in 
evaluation requires a model which relates values, myths and paradigms. 

As early as 1971, Daniel Stufflebeam raised several issues of concern to evaluation change 
agents-issues which have yet to be adequately addressed by prevailing evaluation models today. 
He notes four problenis in the application of experimental designs to evaluation. First, the use of 
experimental designs conflicts with the principle that evaluation should facilitate continuous 
program improvement According to a colleague of Stufflebeam, "experimental design prevents 
rather than promotes changes in the treatnents because ... treatments cannot be altered if the data 
about differences between treatments are to be unequivocal" (Guba, 1969, p. 34). Secondly, 
traditional research methodologies are useless for making decisions during the planning and 
unplementation of a project Rather, the stress on controlling operational variables creates a 
contrived situation and blocks the collection of natural and dynamic information. Thirdly, the 
typical research design does not apply to tiie "septic" conditions of most evaluation contexts. That 
is, the evaluator is not interested in establishing' highly controlled conditions v/ithin which 
universally true knowledge can be generalized. Instead, "one wishes to set up conditions of invited 
interference from all faaors that might ever influence a learning (or whatever) transaction" (Guba, 
1969, p. 33). As a final point, internal validity is gained at the expense of external validity. 
Clearly, equating evaluation models with empirical models is destructive to dynamic program 
development 

In essence, we are considering the semiotics of improvement-oriented evaluation models. 
Porsecanski (1989) captured this dilemma well when he said "individuals and groups do not really 
acceptchangcs that they do not desire or consider necessary, and tiiat tradition, beliefs, and social 
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Status generally prevail over change". This dilemma is further compounded in evaluation when the 
evaluator threatens to question existing structures and commonly accepted myths. 

More on the Literature of Educational Evaluation Models 

The current practice of evaluation relies hec^dly on tiiree models developed over twenty 
years ago. This section proposes to examine these models in terms of three tasks basic to any 
evaluation involving culturally diverse contexts, i.e., how they determine what is to be questioned 
or studied, how they determine and systematically collect relevant data, and how they determine 
operating values and judge program worth. 

Model diversity emerges from the various emphases placed on each of these tasks and from 
the evaluatofs relationship to the client system in regard to these tasks. For example. Pace's 
(1968) analysis of evaluation models indicates that v/hen the total unit tO be evaluated is large, 
complex, and of long duration, a different model is required - one that considers a broad range of 
social and educational consequences. Focusing on the assessment of program objectives must be 
expanded to question other program dimensions (e.g., expected and unexpected consequences, 
cultural characteristics of the setting, the process of program delivery). Evidence gathered to 
answer these questions should be both quantitative and qualitative, including the systematic 
collection of personal perceptions. 

To demonstrate the different emphases placed in various evaluation concerns, the following 
section analyses three prevailing approaches: the instructional objectives approach, the decision- 
making approach, ard the values-based approach. Special attention is directed to the presence or 
absence of several criteria: a systematic methodology which utilizes both quantitative ("hard") and 
qualitative ("soft") data sources to portray a more holistic reality; helpfulness toward program 
improvement; the evaluctor's openness to make values overt and to facilitate the planning of 
probable futures for a program* 

Instructional Objectives Approaches 

Tyler is laiown for an evaluation model which stresses measuring student progress by 
means of instructional objectives. Originated over fifty years ago, he conceiv^es evaluation as the 
process of assessing the degree to which a program is achieving its predetermined objectives. The 
model is not concerned with judgments of the value of these objectives as much as the need to spell 
out the objectives behaviorally. These objectives then serve as the goals for teaching and the basis 
for testing achievement Ideally, measurement is broad in scope and continual. In addition, Tyler 
proposes teaching and evaluating both cognitive and affe^nive outcomes. While his definition 
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focuses evaluation on "results," Tyler's (1942) vision of the purposes of evaluation is broader than 
products: 

a. provision of a periodic check on the effectiveness of the educationrU institution; 

b. validation of the assumptions upon which the educational institution opiates; 

c. provision of information basic to effective guidance of individual students; 

d. provision of "psychological security" to school staff, students, and pgrents; 

e. provision of a sound basis for public relations; 

f. help to both teachers and students to clarify tiieir purposes and to see more concretely the 
directions in which they are moving. 

Using Tyler's approach, an evaluation consists of five basic steps. First, educational 
objectives are form^dated and classified according to the level of required specificity. Second, 
objectives are defined behaviorally. Third, situations are identified in which die behavior could be 
expected to occur. The fourth step involves the technical methodology for appraising student 
behaviors. Once collected, behavioral data are intapreted and then recycled for design 
improvements. That is, intended objectives can be revised Such improvements lead to the 
modification of teaching which in turn leads to anotiier evaluation and revision and so on. 

Tyler's model has several merits. It provides valid, reliable, and objective data for an 
evaluation. It allows the evaluator to indicate attained and unattained objectives. 

On die otiier hand, strict application of die Tyler Model creates difficulties. It ascertains 
student outcomes but ignores die contexts and processes that lead to these final outcomes. The 
statement of objectives in behavioral terms is a long and often tedious procedure which limits 
sources of evidence to purely quantifiable data- In addition, it is too easy to avoid questions about 
die worth of the objectives themselves, particularly if die evaluation is carried out after die 
objectives are formulated. When this happens, die usefulness of the evaluation tends to be limited 
to superficial reformulations of performance objectives. Hnally , the question of comparing various 
programs using Tyler's model is explicitly avoided 

In response to diese difficulties, more flexible "neo-Tylerian" models have emerged (Taba 
& Sawin, 1962; Cronbach, 1963; AAAS Commission on Science Education, 1965). Taba and 
Sawin propose an evaluation model diat focuses on dit collection of information in order to 
determine why ^ome students failed to achieve stated objectives. Tyler's data sources have been 
expanded to include observations on teaching method, patterns of classroom interaction, physical 
facilities, and student motivations. 

"Neo-Tylerian" ideas have contributed to die field*s shift fiom a terminal focus to one 
which syntiiesizes botii process and product elements. The emphasis of evaluation is on facilitating 
curriculum improvement In this audior's opinion, two major limitations persist One, the 
evaluators do not incorporate their clients* values into the proposed evaluation. Secondly, futures 
planning is neglected. 
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Decision Making Approaches 

A second group of evaluation theorists believes the evaluatofs task is one of delineating 
information to be collected, planning a data collection methodology, and helping decision makers 
use the information collected It is the responsibility of the decision maker, not the evaluator, to 
mal:e jndgments from this information. Four questions are basic to this approach: 

a Whai shoiiM the objective be? 

b. How should the program be designed to meet these objectives? 

c. Is the program design being carried out effectively and efScientiy? 

d. Was the program worth what was invested in it considering die products achieved? 
(Reasby, 1974, p. 23). 

Major decisioi. making models to be discussed are: the Discrepancy Model and the CIPP Model 

A. Discrepancy Model «. 

The Discrepancy Model, developed by Provus (1969, 1971) emphasizes the discrepancy 
between die ideal standards in a program and the real performances, Provus assumes tiiat tiiere can 
be no evaluation witiiout a standard The explicitness of die standard generally determines the 
precision of die evaluation. It is die task of the evaluator to assist and train die program staff in 
evaluation. Evaluative training should emphasize continuous and cooperative involvement between 
die evaluator and his clients. Special attention is paid to training the staff in: 1) delineating and 
agreeing on program standards; 2) awareness of political and scientific considerations; and 3) 
determining whedier a discrepancy exits and using diis information in a feedback loop to identify 
program weaknesses. 

According to .Provus, education programs must be evaluated in stages relative to the 
development and stability of die progi-am being investigated Five stages of evaluation have been 
posited in die Discrepancy Model: design, installation, process, product, and cost At each stage, 
comparisons are made and discrepancies sought bttween operational reality and some ideal 
standard derived from the values of the program staff and die client population it serves. 

Provus (1971) concludes diat die major weakness widi die discrepancy model seems to 
stem from failure to understand die limits on institutional behavior set by field conditions, the 
intricacies of the decision-making process, and the origin and use of ciiteria w that process. Odier 
criticisms include: die absence of a context evaluation and the notion diat die use of behavioral 
standards may limit the creative, adaptive responsiveness of a program staff. Using Provus's 
approach is unclear who actively participates in program decision-makmg. Equally unclear is die 
type and ultimate use of information to be collected at each stage. Hie model could also be faulted 
for being incapable of evaluating rapid, larpe-scale changes. Its ability to evaluate more dian one 
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program at a time is also questionable. Decision-makers are not always rational, yet the model 
assumes such behavior. Finally, the Discrepancy Model does not address how evaluators are 
recruited, trained, and supported in the system. 

The CIPP Model 

The Phi Delta Kappan Committee on Evaluation, chaired by Daniel Stufflebeam (1971), 
has perfiaps exerted more influence than any gioup in attempting to lead the educational profession 
away from excessive reKance on classical research models. Instead, a decision-making evaluation 
model is offered which assists decision makers in pinpointing their values so that they can best be 
served by the decisions made. This model, known as CIPP, specifies four types of evaluation: 
Context, iiput, Erocess, and Eroduct (1983). 

Context evaluation defines the relevant environment, describes the.d, ired and actual 
conditions pertaining to that environment, identifies unmet needs and unused opportunities, and 
diagnoses the problems that prevent needs from being met and opportunities from being used. 
Context evaluation serves planning decisions in order to determine objectives. In order to 
"monitor" the total system, tools such as surveys, standardized tests, and demographic statistics 
are used. 

Input evaluation provides infonnation for determining how to utilize resources to meet 
program goals. It involves identifying and assessing three aspects: a) relevant capabilities of the 
responsible agency, b) alternate strategies for achieving program goals, and c) alternate designs for 
implementing a selected strategy. Data sampling methodologies might include committee 
discussions, research of the literature, and consultant assistance, input evaluation serves 
structuring decisions by specifying a plan of action to ac'iiieve desired objectives. 

Process evaluation provides periodic feedback to persons responsible for implementing 
plans and procedures. The strategy involves a) identifying and continuously monitoring die 
potential sources of failure in a project, b) projecting and servicing preprogrammed decisions, and 
c) describing what actually occurs during the program. Process evaluation serves implementing 
decisions in order that project operations can be controlled and the plan of action realized on a day- 
to-day basis. 

Product evaluation's p ^ose is to measure and to interpret attainments. This occurs not 
only at tlie end of a program or project cycle, but as often as necessary during the program. In 
contrast to a terminal or "product" focus in Tylerian models of evaluation, the CIPP model 
considers product evaluation as a continuous activity since attainments occur throughout the 
program. Product evaluations specifically serve recycling decisions in order that project outcomes 
can be reacted to and judged. Furthermore, it serves tlie decision maker, who must decide whether 
to continue, to terminate, or to modify a pr ognm. 
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The procedure for the four evaluations mentioned above is basically the same, Firsdy, 
identify the type or decision to be serviced. Secondly delineate the information needed. After the 
relevant decision makers have been identified, develop techniques to obtain and collect this desired 
information. The final step involves providmg the information to relevant decision makers and 
audiences. 

Though mechanically total, the CIPP model excludes overt values in its schema. 
According to Stufflebeam, the evaluatofs role is one of collection, organization, and analysis of 
"relevant" data for "relevant" decision makers. The evaluator is not directiy involved in program 
decision-making. Overt judgment of the intrinsic worth of the program's objectives is not 
considered by the evaluator or the clients. The entire question of values is kept tacitly in the 
decision-makers' domain. When the CD?? evaluator provides infomiation to the decision-maker 
regardiiig which values will be served by alternative decisions but does not offer overt judgm.ent 
himself, -he is demonstrating the logical inconsistency of CIPFs theoretical stance on values. 
Though tacit, the selection of methods to uncover these values is largely a matter of the evaluator's 
values. Another limitation of the CIPP model is its inabilit}' to answer three basic questions, 
Firsfly, how do cvaluators and/or clients know which values are operant? Secondly, how can 
various effects of schooling be determined? I^tiy, what processes are necessary to enable the 
decision maker to apply value criteria? 

Values-Based Approaches 

There are two major examples of values-based approaches to program evaluation. The first 
to be discussed is Scriven's Goal-Free Model, The second model to be discussed involves Stake's 
responsive schema, 

A. Goal-Free Model of Evaluation 

Scriven's (1967; 1972; 1978) Goal-Free model is in definite contrast to evaluators 
advocating objectives-based or decision making approaches. According to Scriven, the distinction 
between the roles and goals of evaluation are often intentionally blurred Whatever its role, the 
goals of evaluation are always the same - to estimate the merit, worthy or value of the thing being 
evaluated, Scriven goes on to point out that the subversion of goals to roles is very often a 
misguided attempt to allay the anxiety of those being evaluated. The consequence of this kind of 
distorted evaluation could be much v^ort undesirable than the anxieties evoked, 

Scriven divides evaluation into two areas, formative and summative, Fonnative evaluation 
is concerned with questions such as : What goals should the program achieve? What is the plan 
for achieviM these goals? and Does the operating program, when developed and put into practice, 
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achieve the desired goals? Summative evaluation comes at the end of a program. Basically, it is an 
assessment of the finished product 

The distinction between summative and formative has implications for the personnel 
involved in the evaluation. The formative evaluator must work closely with the program manager. 
For the summative evaluator, quite the opposite is true. He must be free of any potential conflict of 
mterest so that his evaluation has an integrity of design and conclusion, 

Scriven (1972) declared that both summative and formative evaluations will increasingly 
become ''goal-free," Believing intermediate god evaluations to be unnecessary, Scriven stresses 
the evaluation of actual effects against a profile of demonstrated needs. One of the many roles of 
the evaluator is to examine the goals of the educational program and judge the worth or value of 
diese goals against some standard of merit From the data he has collected, the evaluator can 
determine whether these objectives are being met Formative value judgments of the program are 
made for producers and summative value judgments are made for th^^ consumer. 

On many occasions, the staff evaluator is likely to have "tunnel-vision which respect to the 
effects of the materials (or methods, etc). That is, a tendency to look mainly in the direction of the 
announced goals" (Scriven, 1972), Therefore, it is essential that there is an external evaluator who 
has not been informed about the project's goals so that he may devote his xmbiased attention to 
looking for actual effects or side-effects of a project Thus, he may notice something that everyone 
else has overlooked or he may, prouuce a "novel overall perspective". 

Several questioas remain unanswered about "goal-free" evaluation. One major concern is 
how best to insure that external evaluators properly judge actual effects of a program, whether 
planned or not. What standards are there to judge whether a goal-free evaluator is not arbitrary, 
inept, or unscrupulous in his actions? How does one judge how well a goal-free evaluator has 
interpreted the "demonstrated ne^" of a project? It seems these questions will not be answered 
until Scriven sets up the necessary dynamics of mutual negotiation between the evaluator and the 
evaluated Only in this way can he realize a holistic evaluation schema that includes both internal 
and external values. 

Bo Stake's Model of Evaluation 

Stake would undoubtedly agree with Scriven's feelings about goal-free evaluation. The 
theme tfiroughout much of Stake's writings (1967, 1970, 1972, 1973, 1975, 1982) is that an 
evaluator must do his best to reflect the nature of the program and not focus on what is most easily 
measured. Furthermore, he writes less about the link between evaluation and decision making than 
other evaluation theoreticians. 

Stake agrees with Scriven that both descriptions and judgments are essential and basic acts 
of any evaluation and should be combined to portray an educational program. For Stake, the rote 
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of the cvaluator is much more than an interpreter of test scores. He beKeves that the evaluator's 
task is one of identifying outcomes or "intents" that arc contingent upon particular antecedent 
conditions and instructional transactions which occur between teachers and students, students and 
teachers, students and students, etc. These contingencies seek to discover the connecting 
relationships between transactions on two levels: first, what was intended; second, what is 
observed In addition to contingencies, the evaluator examines the congmency of these intents and 
observations in terms of antecedents, transactions, and outcomes. Thus, descriptions are obtained 
through a process of presenting a rationale for the evaluation based upon the local situation, 
followed by articulation of intents and observ£.tions. 

Stake also believes that the role of die evaluator is one of making subjective decisions in 
selecting variables for the evaluation. These judgments are reached by having "experts" generate 
standards for antecedents, transactions, and outcomes. Than, intents and actual observations are 
compared with these prescribed standards of excellence. Finally, die evaluator makes judgments 
of worth based upon these standards. Thus, the evaluator should be a specialist in die collection, 
processing, and interpretation of data^ Moreover, she should be an expert in die presentation of 
data, to the client system. She should be familiar witii other programs having similar objectives or 
intents, and she should provide the client- system witii tiiese comparisons. Stake stresses that data 
be collected from a wide variety of sources and that die evaluator's role is to assist the educational 
enterprise in a realistic formulation of objectives and in the assessment of the congruence of these 
witii die observable data collected The evaluator is a technician who can provide both relative and 
absolute judgment In tiiis respect, Stake takes a more conservative stance than Scriven, Stake 
(1973) recommends that die evaluator should not take an absolutist position regarding die 
program's goals since this is likely to make clients less willing to cooperate in die evaluation. 

The Stake model exceeds Tyler's model in :ittempting to describe and to judge die entire 
educational enterprise, ratiier tiiaji examining outcomes alone. In terms of process and scope, die 
Stake model deals more comprehensively witii a number of evaluation needs. The assessment of 
evaluation procedures in die desaiptive matrix is unclear as it appears tiiat procedures would be 
described with transactions. However, procedures are selected prior to the learning experience. An 
effective meUiod for analyzing the role of values in die program can be found in tiiis model as can a 
mediodology for discovering which values are being served 

Like Tyler, Stake does not pro\dde for evaluating decision alternatives during die 
structuring of the learning experience, Neitiier does he provide for adequate feedback in program 
development. While die educational program is looked at in terms of antecedents, transactions, ind 
outcomes, underlying die whole approach is a perspective diat looks at evaluation fitom die 
terminus. It does not forcefully facilitate dynamic development of educational programs. 
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The Neec .or a Theory of Evaluation With Explicit Quality Criteria 
It appears current evaluation models can aeate methodologically sound strategies but 
credibility tends to be lacking unless some process exists for making the operational values of both 
the evaluated and evaluators overt Furthermore, these overt values need to be incorporated into 
the evalu^ttion itself Support for this position has been persuasively advanced by Weckwonh: 

"First, there is no one way to do evaluation; second^ there is no generic logical stnicmre 
which will assure a unique "right method of choice/" Third, evaluation ultimately becomes 
judgment and will remain so, so long as there is no ultimate criterion for monotonic 
ordering of priorities; and fourth, the critical element in evaluation is simply: who has the 
right, i,e,, die power, tiie influence, or the authority to decide/' (1969, p. 48)* 

It is die purpose of tiiis section to offer a theoretical nnxiel which strives for credible 
syntiiesis-which is especially difficult to achieve in contexts reflecting diverse client and audience 
values. It assumes that there arc no theories - only personal perceptions of theories (Polanyi, 
1966) and tiiis is what will keep the model dynamic • A model is thereforc perceived as !\ body of 
inteirelated criteria, which in the case of the prcpused model are quality criteria. These (quality) 
criteria present view of a topic. The topic's focus is a function of the builder's personal frame of 
i.'eference. Specifically, perceptions and judgments are a function of the a^'.thofs Umwelt. By 
making overt die v^alnerable judgments of bod^ ^valuators aiid diose being evaluated, a syndietic 
picture of what die evaluation intendf; to foc*'^ on can be negotiated, Thiough negotiation, 
operative values can be embodied in the major evaluation questions asked. In addition, diese values 
can serve as judgment criteria for decision-ixiaking. 

In die perception-based model of evaluation proposed, certain value criteria are not 
negotiated in die sense diat along with accepting die evaluator personally, die f .imary audience 
must accept five quality criteria inherent in the model That is, die model embodies the following 
criteria: holism, helpfulness toward program improvement, evaluator vulnerability vnui die 
audience, acceptance of bodi *Tia?:d" and "soft" data sources to portray reality, and facilitation of 
planning a program's future. Additional quality criteria are negotiated widi individual audiences. 

Five Evaluation Signs 

Analysis of perception-based models of program evaluation suggest five core "sigi.^" 
which advocates perceive as essential to conducting and interpreting studies. In fact, die first four 
assumptions of DriscoU and Kynn's research semiotic are very congruent widi die first four 
perception-based signs delineiated below. 

First, a holistic or comprehensive base is advocated rather than the 
reductionistic principles frequently associated with western dominated program 
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evaluation designs. Education and training are considered broader, socio-economic 
phenomena which require a synthesis of multiple perspectives, including psychological, social, 
political, anthropological, philosophical, and economic. Further, humans are regarded as creative, 
historical forces whr in need of a critical, reflexive way of knowing (Alio, 1961; Freire, 1970 
a,b). Man is real and active to the extent that he perc«v€5 md produces. Active men are 
timebound and are conditioned by a determinate social structure and naturs, 

Theoiy should be holistic rather distort the total evaluation pi;.ture with an undue emphasis 
on quan'dfication. That is, it goes beyond mere correlational analysis or statistical inference witii tlie 
addition of an existential phenomenological^haosophical treatment wherein all syntiieses are 
repeatedly tested against e^qyeri^tial pCTceplions, 

There is no way of surveying die whole of reality since one has no assurance that he has 
experienced anything more tiian an infinitesimal san^le of it As different metiiodologies 
increasingly seek to make reality determinant, tiiey become caught in their own closed system. A 
labyrinth of tiieories is constructed in an effort to reduce the excluded aspects to the included ones, 
but there is no q)ening in this finite maze. For example, the natural sciences have limited the 
conception of nature to that which can be empirically investigated, Le,, the mere description of 
observable experience. We do not object to die scientific method, but only to die regarding it as 
exhaustive of cognitive metiiod 

Support for this particular criterion is especially strong in die work of Prcnch change 
dieorist, Georges Alio (1961). Science, claims Alio, necessarily fi:agments reality when it 
abstracts. He challenges this procedure in die specific case of value studies because by definition 
values involve evaluational totality, not mferely valuational fiagments, A normative sequence which 
delineates existing value constellations and explores alternatives in terms of probable future is a 
result of Allo's examination of third world contexts. One can see parallels in Allo's research 
metiiodology to die pedago/jy of die oppressed devised by Paulo Freire (1970a), Bodi have 
dkected their efforts towari the reduction of etiinocentrism and attempted a holistic orientation by 
means of testing knowledge through action which is critical, dialogical, and reflective. 

Second, vs^bMe promo^irig the value of commitment toward program 
improvement versus proofs inherent in objectives-based evaluation models, it is 
recognized that practitioners of this approach can sometimes experience difficulty 
avoiding alienation of diverse constituencies (e.g., funders, practitioners, 
informants). Evaluators must also be eclectic and capable of listening to divergent values so that 
a truly "syndietic" base of program improvement can be perceived as helpful radier than imposed 
Negotiation among all the key constituents (Le., trying to achieve 360 degrees of representation) is 
advocated throughout the evaluation process, including ne^tiating the initial evaluation paradigm 
and its major questions of focus. 
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The negotiation should be helpful in terms of facilitating the commitment of all parties 
concerned to ^ * ^gram improvement The values of the evaluator and the evaluated are exposed by 
overt, dynamic dialogue, negotiation and pedagogical action. By placing responsibility for the 
change effort directly in the hands of the evaluated, it becomes increasingly probable that these 
change pedagogics will enable the understanding of a system's relative value positions in their 
totality. Reconstruction or improvement of the outer boundaries of a "society's existence 
rationality" can then result while its "core values" remain respected and intact Overall, this 
negotiation process becomes e-spedally challenging when the evaluatofs "core values" differ fix)m 
his clients, sources or consumers. 

Thirds avoid a reduvctionistic "resc irch^' perspective due to the inherent 
danger of discredSting obvious localized learning and ^rass roots data in favor ou 

hard numbers". Normative or criterion-refaenced appro' *es dominate and there is no 
question that trying more synthetic models is a risky business-politically, economically and 
theoretically. If we assume this risk, creative divergence gained from a unification of qualitative 
and quantitative sources of evidence (as opposed to data collection strategies of control theories 
such as Tyler^s) becomes an actualizing component of potential realities rather than the status quo. 

The model can then duel with the positivist and reductionist biases inherent in empiricism 
by granting primacy to tiie realm of synthesis and the totality of experience. The limitation of 
analysis can transcend a theoretical framework of deductive hypotheses and quantification as the 
only test of truth. Abductive and inductive frameworks can be included in the model. As Goulet 
(1971) has cautioned: 'It is true that quantitative changes of a sufficient magnitude may produce 
changes in quality, but even in these cases, the uniquely qualitative dimensions of reality tend to 
esc^e the purview of those best equipped to measure or describe quantitative facets of a whole. 
Consequentiy, one may reasonably doubt the ability of social scientists to perceive qualitative 
manifestations of value change except through the distorting lens of their own positivist bias (p. 
219). Thus, for our purposes, abstraction and analysis are applied only at certain moments in a 
specified sequence. 

Fourth, the concept of evaluator vulnerability is perceived positively rather 
than as something to be defensively covered up. Vubierability is used in die sense of 
actively seeking multiple perceptions of key actors, whether in context, process or product stages 
in order to achieve the very unstead'^ state of intersubjectivity. Evaluators are seen as objective 
ratiier than neutral (neutrality being equated with stasis or death). Basically, it appears 
vuhierability can be characteristic of an effective change agent, but the premise does not extend as 
far as Machiavelli's end of the continuum witii his contention that this openness is a necessary 
illusion. Pharies' (1985) interpretation of semiotic reality as "die final setded opinion of an 
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indefiiite community scientific investigators" would be eschewed as elitist and a more open 
systems idew of shareholders included in the process of defining and judging a program's reality. 

The theory should deal with the elitism of the evaluator as the possessor of distinct skills 
and a quasi-mystical database in order to avoid manipulation of the evaluated In his exploration of 
the assumptions of Westem social scientists, Goulet has discussed the damaging ramifications of 
elitism. He contends that "researchers practice a new form of clericalism — not of ordained clerks, 
but of methodologists initiated to the exclusive task of explaining reality's crucial dimensions. No 
matter how insistently an individual researcher disclaims elitist views, he is in fact the possessor of 
a mysterious tool-kit, the contents of which cannot be fathomed by the 'objects' of his study. The 
stance of the researcher manifests something of the 'magical' posture described by Levi-Strauss 
(1969, 1976, 1979) when he speaks of the power which accraes to anthropologists investigating 
primitives who lack mastery over the written word. 

By recognizing that one's perceptions confirm, continue, enlarge, and eventually, correct 
one another, it is possible for the evaluator to synthetically enter into the perceptual systems of the 
evaluated. Disagreements may well occur. These are removed, at least they are considered as 
removable by discussion and mutual criticism. According to Edmund Husserl (1950) the 
consciousness of a common perceptual worid as the universal horizon within which all of us live 
arises through intercommunication. "I may and do perceive the same thing as my fellow man, but 
the same thing presents itself to each of us fix)m a different side, under a different aspect and 
perspective. Every such perceptual presentation implies in its inner horizon references to further 
perceptual presentations, all belonging to one total system" (Gurwitsch, 1966, p. 430). By being 
open to the multiplicity of perceptual experiences of both his own and liiose being evaluated, the 
evaluator is not trapped in a narrow kind of consciousness of egocentrically determined utiKtarian 
material. Recognition of the potential of intercommunication in contributing to an accurate picture 
of reality has been articulated by Aldous Huxley (1963) in The Doors of Perception. 

"Every individual is at once the beneficiary and the victim of linguistic tradition into which 
he has been bom — the beneficiary inasmuch as language gives access to the accumulated 
records of other people's experience, the victim insofar as it confirms him in the belief that 
reduced awareness is the cily awareness and as it bedevils his sense of reality, so that he is 
all too apt to take his concepts for data, his words for actual things. That which in the 
language of religion, is called 'this world' is the imiverse of reduced awareness, expressed, 
and, as it were, petrified by language. The various 'other worlds,' with which human 
beings erratically make contact are so many elements in the totality of the awareness 
belonging to Mind at Large" (p. 23). 

To paraphrase Merieau-Ponty (1962), language is v^r^t>c^e/z with our horizon upon the world 

and humanity. Language is borne by our relation to the world and to others which in turn supports 

and creates it It is through language that our horizon becomes open and 'endlos.' Thus, only by 

developing the evaluation in a dialogical way so that both the evaluated and evaluators participate 
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actively in overtly defining the ground rules of program improvement can manipulation be 
prevented and an accurate picture of reality articulated Only through commitment to seeking 
multiple realities of truth can the evaluator be a responsible decision-maker and future plannen 

Finally, a positive view of futures as part of an evaIuator*s 
Weltanschauung is maintained. Sebeok's notion of behavior being modulated by one's 
immediate and past environments is recognized, but perception-based evaluation consciously 
develops a sense of futurism as an essential truth. In essence, we are referring to Peirce's notion 
of a sign of possibility which is represented by its interpretant Unfortunately, present "realities" 
are often used as an excuse for inaction* One step in pulling evaluators fix)m this abyss could 
include two interpretive strategies culled by Driscoll and Flynn's guidelines for conducting 
educational research: presenting evidence in ever expanding netwoiics of implications (Campbell, 
1984) and making only noodest pronouncements given truth is not fixed and making judgments of 
usefulness or effectiveness within interpretive contexts (Cunningham, 1987)* 

The topic of probability offers a further polemic when one considers the significance of 
judgment as a "willed reality," One can argue that empirical research is based on a probability of 
truth, while its coimter, evaluation, values individual perceptions as central sources of data and 
judgment Probability plays a key role in both evaluation and educational research by developing a 
game of signs. Both evaluation and research follow certain mles and are contained within certain 
boundaries which create a saise of order* Players in each game like to delude themselves with the 
belief they are operating in a realm independendy ot extraneous variables, i.e., independently of 
"ordinary" life. The players flaunt certain credentials as possessors of an exclusive information 
base. Educational researchers more blatantly sacrifice meaningful data to reductionistic aiid 
poativistic analysis. Program evaluators who adopt the researcher game plan are more subtle and 
ultimately more distorting of the "ordinary" reality they attempt to capture and judge. Evaluators 
must be careful not to confuse the characteristics of a good evaluator with a good magician. 
Drawing upon the work of the magician Blackstone, Baron (1987) lists seven mles for good magic 
which program evaluators have been know to abuse on occasion: 

• use conversational skills to achieve misdirection 

• play on audience's presuppositions 

• play on audience's expectations 

• never tell an audience how a trick is done 

• the best tricks are those in which the audience has an active role. 

• the effect is what matters, not the means of achieving it 

An empirical sign only partially reveals its dynamic object and the extent to which the 
dynamic object is revealed is the immediate object The dynamic object is what actually determines 
the sign. It is the essence of the program we seek to judge. How does the evaluator see beyond 
the inamediate object to the dynamic object? It is the dynamic object that has determined the 
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descriptions but the revealed descriptions are not sufficient in tiiemselves forjudging the dynamic 
object The evaluatcr must depend on "collateral experience to grasp the true significance of the 
sign and make the correct diagnosis" or judgment (Houser, 1987, p. 265). 

Another distinction between these evaluation and research signs can be seen in 
empiricism's tendency to develop as a closed system directed toward refinement of previously 
stated objectives versus holistic program evaluation's proactive orientation to die novel and open 
future. To the empiricists, research outside of the positivistic paradigm is valued only after a full- 
blown crisis destroys any viability for the established principles of the scientific method (Kuhn, 
1962). Cunentiy the reactive nature of empiricism is shakmg the stmcture of science, witii the 
resulting realization that future planning is important and "possible in fields lying outside of die 
bounds of absolute scientific causality . . ." (Gordon, 1969, p. 73). Thus, a proactive evaluator 
must base his conclusions and recommended course of action on his judgment of an improved, 
better willed reality. 

An Evaluation Seniiotic: Perceptions and Polemics 

Evaluation involves tiie interaction of signs, objects and inteipretants. Evaluation questions 
are the tools that enable us to present unfamiliar, change-oriented descriptions and judgments 
(dynamic objec's) in terms of familiar, accepted descriptions and judgments (inmiediate objects). 
Descriptions and jur;gements are grounded in perception. That is, perception is considered the 
central source of evidence which permits the interpretation of data in terms of systematically 
ordered concepts. In a specific sense, perception is defined as what individuals report as their 
descriptions of events, activities, and values. Quantitative definitions of perceptions such as one 
finds in neurological and sense datum expositions have been found to be limited for our purposes, 
and the necessity of a definition incorporating Polanyi's (1966) 'tacit dimension* has become 
increasingly apparent That is, "we have seen how the quality of die senses are both determinate 
and indeterminate, and, tiierefore, how a reality is directiy present to consciousness, but present 
only partially. Such is the character of die world apprehended by subjectivity." (Earle, 1955, p. 
99). Bergson uses the metaphor of a perceptual chain: "we can . . . conceive of succession witiiout 
distinction and think of it as a mutual penetration, an interconnection and organization of die 
elements, each one of which represents die whole and cannot be distinguished or isolated fi:om it 
except by abstract tiiought" (1910, p. 101). Schutz (1967) has expanded Bergscn's concept of 
consciousness as an on-rushing stream of prereflective experience, which man lives through ratiier 
tiian knows about, and which is tied to some concept of time. The similarity here with Merleau- 
Ponty's imperative to philosophy to return to perception, tiiat is to prcreflective and prescientific 
experience is considerable. And die similarity increases when Schutz's accounts of die 
understanduig of otiier people, and die social world in general are considered And lUst as die 
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analysis of perception is, forMerleau-Ponty, analysis of the world-as-perceived, similarly for 
Schutz, analysis of subjective social knowledge is an analysis of the social worid-as-known. For 
bothphenomenologists the reference of philosophical analysis to the subject involves necessarily a 
descriptive and analytical recapture and reconstruction of the objective world, the objective 
correlate of subjectivity. 

Moving theories into action results in paradigms. As Kuhn said (1970), die scientific 
paradigm is like a common door through which scientists must pass in order to understand their 
world and design new research. But most of these researchers are not conscious of the 
reductionistic assumptions imbedded in their paradigm, nor are many evaluators. Evaluation 
paradigms or myths consist of dynamic interactions between predictive and polemical relations. 
Signs in evaluation must take into account questions (contextual, process, product), sources of 
evidence, and qualiticg/standards by which to judge the resultant data and questions. These 
connections between questions, sources and qualities are not explained by real objects but rather 
perceptions of connections and objects. According to Riffaterre's (1988) notion of hypersigns, 
this cormective process occurs through extension or deduction from a prior sign, which has the 
effect of imposing a re-reading and meta-mapping of that sign. For example, when we look at the 
connection between evaluation contexts and evaluation products, we need to examine both 
anticipated or ordinary signs and unexpected or extraordinary signs. 

At this point it should not be for^jor^n that a person does not report a 'purely objective' 
description, only good or bad, better or mxse descriptions which involve selective and systematic 
distortions. An evaluator must consciously deal with the fact tiiat individuals selectively perceive, 
interpret, and place meaning upon tiieir world, and tiien act accordingly. In fact, the solid facts of 
society are tiiese perceptions, definitions, interpretations, and meanings which enable the perceiver 
to operate effeclively most of die time in die face of incomplete informational input or under 
conditions of impaired observations. Franz From (1971), a Danish psychologist, believes 
perceptions of another's actions depend upon our familiarity with such factors as the type of action 
sequence played, situa. ional norms, the actor or persons of his type, and whether we knov/ 
sometiiing about how tiiis kind cf man usually behaves. As our familiarity witii these factors 
becomes mcreasingly vague, "we see that others do something or other, but not what they are 
doing, or die experience becomes very disjointed without any connection or transidon between die 
details. There may in such a case be given a more comprehensive, ratiier imprecise, sense." (p. 
91). 

The intention, even beyond our explicit thoughts, h?s been to open the sphere of reality to 
whatever we know and tiierefore, beyond the constrictions of quantifiable sense peiceptions of 
physical substance. This view permits as 'real* the concepts of subjectivities, selves, and 



ERIC 



IS 



AN EVALUATION SEMIOTIC 17 



personaKties. All have a mode of being distinctly peculiar to themselves, and reducible neither to 
physical existence, biological life, sheer duration, nor to instantaneous actuality. 

Li a general sense, perception refers to what someone judges in terms of the overall 
theoretical construct. Judgment is viewed as the subjective act which wills something as true, in 
effect, a "willed reality" (Earle, 1955, p. ; SD) which usually occurs in the sphere of the 
indeterminate where probabilities concern* ^^g the truth of realities operate. Judgments are 
considered real only to the extent that a person's reports truly reflect his perceptions. It does not 
have the tmtli-revealing character of apprehension. Judgment thus is a very secondary sort of 
thing, completely dependent upon perception for its confirmation. Problems in program decision- 
making oorvj when something, someone, etc., is incapable of bemg direcdy perceived, in which 
case only a vague report of the operating perceptions can be gathered Then, judgments about their 
particular qualities and their mode of being must be grounded upon not a single report, but upon a 
number of perceptions-descriptions fix)m a number of subjects. 

In considering the need for judgments in program evaluation, a more extensive examination 
of judgment is necessary. Tlie concept of judgments as a "willed reality" evolves from imagination 
and personal valuing. Drawing fi-cm die actual world and provided by experience, imagination is a 
way of discovering and expanding existing potentialities, i.e., to really stand under tiie visible and 
direct and obtain meaning. The extent tiiat possibilities revealed by imagination can go beyond die 
superficial and trivial, tiirough die dynamic and creative powers of tiie evaluator, determines die 
extent of utilizing tiie deepest character of the program evaluated and expanding the arcna of the 
new "willed reality," or vision of die better; die greater die range of major questions asked, die 
more legitimate die evaluatof s judgments in making the third box and its willed realities the 
possibilities of odiers. This component, diereforc, becomes an actualizing component of potential 
rather dian a component of die description of status quo. As Polanyi has stated ". .. to see a 
problem and undertake its pursuit is to see a range of potentialities, believed to be accessible." 
(1966, p. 37). 

What seems to be operating here are levels or potentialities determined by degrees of 
imagination. Indecision and lack of active change seem to be based on man placing on himself all 
kinds of limitations to action, analogous to the instincts of animals that determine the behavior 
sequence in advance. Specifically, that which is in accordance widi the rules is good and acceptable 
because it gives peace of mind, while actions which contradict die laws are bad and undesirable 
because diey are unpredictable and create uncertainty. This unrealistic orientation accords primacy 
to as objective and empirical a world as can be controlled, and bans the place of words like 
subjectivity, imagination, tacit knowing, and intersubjectivity. In reaction to diis behavioristic 
tendency, die perception-based evaluation semiotic as presented in this paper has endeavored to 
syndiesize die overt judgments of those operating as evaluators and die tacit value stance of diose 



ERLC 



19 



AN EVALUATION SEmOnC 18 



being evaluated Furthermore, tlirough a formative negotiation process, predetermined operative 
values can be embodied in the major questions asked and evidence sought Finally, an improved 
"willed reality" can be planned 

Paradigm of the Perception-Based Evaluation 

The total unfolding nature of perception-based theory can be seen in the three column 
paradigm in Table 1. The paradigm calls for identification and negotiation of major questions to be 
a^ked (see Column 1). This might include ootii tiiose cited or intended and accidental objectives. 
The second column of die paradigm attempts to describe an "accurate picture" of the program, 
ideally allowing reality to be described holistically by otiiers and mutually agreeing upon tiiese 
descriptions. The third column articulates die values and die rationale for making judgments. In 
fact, inherent in all three columns are five signs: holism, helpfiilness, evaluator vulnerability, 
acceptance of botii quantitative and qualitative dimensions of reality and a vision of die fiiture. 
Negotiated judgments arising fiiom the evaluatra's and participants* egocentric value orientations 
must be made overt in die process of determining major questions. In describing an accurate 
picture of reality, some judgments in terms of validity and reliability are involved. The third 
column synthesizes tiiese elemenfc: by providing a meaningful perception orientation upon which 
holistic judgments about programs are based 

Stepping from opinion and riietoric to systematic evaluative inquiry forces a duelling with 
the description versus judgment polemic. Assuming all thought is in signs, what operates is a 
multi-tiered sign system dependent upon interpreting die descriptions of all die critical stakeholders 
in die evaluation. Sign fluency appears to involve balancing two dimensions of perception: die 
active (i,e,, low filter, holistic, tacit, iconic) and the passive ( i,e,, high filter, part, explicit, 
verbal). 

To make holistic judgments based on die facile reading of diis multi-tiered sign system 
requires die initial step of negotiating major questions to focus die evaluation inquiry witii 
stakeholders; "interaction foregrounds similarity" (Ricoeur, 1985), 

Different tiian non-negotiated research questions, negotiated questions serve as indicants 
and metaphors, "Indication,,.seems to be a way die mind operates in complex states of affairs. 
Through indication we communicate what we cannot conceptualize. As a mechanism for 
indication, metaphor expresses intuition-mental consequences of perception which one cannot 
formulate with die words and syntactic constructions of one's language. The sense of die whole is 
simple. If "the poet can find metaphors that render diese obscure parts accessible and eventually 
understandable" (Benzon & Hays, 1987, p, 60), tiien die evaluator must acquire similar skills to 
render the obscure accessible and be able to perceive bodi the focal and residual elements of a 
question. Metaphor leads to recognition and training in reading signs in first one's own culture 
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and then simultajieous Mendings of multicultural metaphors should lead to a heightened awareness 
in other cultures. Specificail3% Driscoll and Dempsey's (1988) application of the rational set 
generator may be one way to explicate cultural metaph^s at increasingly complex levels* 

How can the same sign (of possibility, fact or reason) mean so much more to one evaluator 
than another? Houser (1987) accounts for different interpretants based on the inunediate objects 
(e*g., sources of evidence) being different Understanding the inmiediate object depends on 
previous interpretants of previous signs-that is, background knowledge. Making judgments 
about the data depends on predefined standards or qualities and background knowledge. If our 
immediate objects are primarily quantitative, our judgments will convey a decidedly empirical 
flavor. Empirical conveyance is an analytical process whereby clarity is dissected and a scientific 
paradigm dismantied into ontological and methodological assumptions. 

More holistic evaluation paradigms must examine what standards or qualities of "good" 
evaluation are. Prior efforts to predefine "good" evaluation processes are signified by die 
following words, many of which encompass but are not limited to reductionistic evalution models: 



broad-based 

reinforcing 

analytical 

critical 

objective 

process-oriented 

negotiated 

site-specific 



mtegrative 
choice 

developmental 

dialogical 

subjective 

product-oriented 

diverse audiences 

realistic 



multi-disciplinary 

personal 

challenging 

systematic 

intersubjective 

planned 

multiple clients 

improvement 



comprehensive 
open-ended 

promotes self-understanding 

affective & cognitive 

ongoing 

futuristic 

accessible 

problem-solving 



By guaranteeing intersubjectivity, (which incorporates objectivity and subjectivity) tiiis 
paradigm can actualize itself in the language of individuals' personal and holistic descriptions, 
observations, perception grounded decisions as to what to inquire into, and recommendations for 
modifications. This language recognizes both die limiting and expanding nature of man's 
conscious interactions. At a prereflective level of consciousness, man can be observed as living- 
tiirough an onrushing stream of undifferentiated experience ratiier than knomng about reality 



insert Table 1 about hat 



ERLC 



21 



ANEVALUAnONSEMIOnC 20 



(Husserl, 1964; Medeau-Ponty, 1965). However, while the subject cannot grasp his own 
immediate flow of experience in the present^ the paradigm accepts Schutz's (1967) argument that 
another self (possibly the evaluator) £an meaningfully catch the other self s present flow o2 
experience. In Beigson's terms (1910) there is a simultaneity between my stream of 
consciousness, my lived experience, and yours. I experience your spontaneous experience direcfly 
as you talk, for instance, but you can only capture your own spontaneous after it has flowed away 
In a self-conscious reflective turn. 

The import of Beigson's and Schutz*s ideas lies in the often neglected communication 
concept that our knowledge of others can be more direct than our knowledge of ourselves. The 
implications of this concept of intersubjectivity on paradigm can be seen in the expanded potential 
of meaningful reality assessment, both by the evaluator and the evaluated Only by discovering 
and makmg evident tlirough language both the evaluated and evaluatof s presuppositic!:s about 
valuing and meaning can die evaluation proceed Mthout going into tiiis subject further, it can be 
seen tfiat Garfinkel's (1970) concepts of "indexicality" (i.e., all language and meaning uepends 
upon conversational context and upon the interpretive work speakers and hearers do in conversing) 
and "reflexivity" (i.e., it is die ongoing attempt in conversational contexts to specify die meaning rf 
those contexts, to formulate what die conversation is about and what is its purpose) speak to the 
dialectical relationship between man and language, between consciousness-purposiveness, and its 
linguistic form and content 

The evaluatkn case study 

The following evaluation case study was selected because it exemplifies perception-based 
tenets. It als^ has die added feature of including contexts where key actors (e.g., funding agency 
staffj trainers, trainees, most odier key informants and die evaluator) are exposed to cultural 
systems which depart from tiieir own. Increasingly, semiotic experts have supported cultural 
interaction as a significant mediator in sensitizing individuals to perceive signs. 

Since 1982, this bi-national foundation has funded nine projects for an average of tfiree 
years each in order to: 

• build a regional network of precoUegiate educators trained to incorporate US-Japan 
relations into die curriculunoi; 

• develop and make quality instructional materials available witii each region; 

• design model training and materials development centers operated by state-based teams. 

The foundation contracted with die fii^t autiior for its extraial evaluation. Conducted over 
a seven month period (May tfirough December, 1988), major questions as well as standards of 
quality were negotiated between die evaluator and die funder. These major questions addressed 
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three phases of the teacher training program: context, process and product Specifically, six 
questions were asked generically as well as other site-specific questions: 

L What are the piuposes and unique characteristics of each center? 

2. How have training and materials been developed overtime and across centers? 

3, What are factors are contributing to ongoing success and failure? 
4 What changes are needed in the overall program? 

5, How effective have the centers been? 

6. What policy issues have emerged? 

Diverse sources of evidence were used to describe each question. Data sources included: 
review of project proposals; foundation project officer files and curricular/traming materials 
developed by each center, analysis of internal evaluation data developed by center personnel; 
visitation to 24 sites throughout the US regions.and four study groups in Japan; individual and 
group inte rv^ewing witfi 154 teachers, 128 smdents, 20 staff, 36 embassy and business leaders, 
seven advisory board members, 11 staff in other foundations, and four foundation staff. 

The evaiuator must analyze parallel and convergent signs, collect descriptions of the same 
event in American and Japanese seniiotic systems, and make judgments regarding the paradox of 
simultaneous decoding of discrepancies and incompatible meanings. The speed of cultural change, 
the impact of technology and the inescapable lure of comparison between the two semiotic systems 
makes sharp analysis difficult Both societies have witnessed tremendous changes, overt and 
subtle, since World Wai ^. For exdinple, what standards do we use to frame judgments about the 
extent of cultural awareness b'ainmg, a goal all tiie funded centers have in comn.Dn? For our part, 
we will seek to apply core values according to Allo's (1961) suggestions, 

A good place to start the analysis is to examine several foundation sponsored summer study 
tours for overt and covert signs of cultural awareness. Imbedded in the travel plans are evidence of 
the degree of "cultural awareness" of its developer as well as the presence of stci;??o'cypical 
experiences versus deliberate tests of tourist rituals that challenge cultural assumptions. 

To approach cultural awareness, one must be able to look at discrepancies of events 
between two cultures. Initially, this is at an implicit level, Benzon and Hays (1987, p, 64) talk 
about "the initial discrepancy between basic global physiological schema Kads you to consult a 
representation which is built upon physiological parts and parts of parts, one which is a 
prepositional construction of these physiological parts. You then search for parts of parts where 
tiie discrepancy is the greatest, tiiat will tell you what has changed" An example of this is the 
discrepancy between pre-tiip images of a country and post-trip images. 

The second phase of developing "cultural awareness" demands an explicit awareness of 
assumptions embodied in one's own culture and anotiier. What about those situations which are 
not aware of a discrepancy because you have been so captured in your own culture? We posit tiiat 
awareness appears to be partially explained by Piagefs notion of tiiat die readiness to perceive and 
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create inner representations of objects and events »drawn horn the external \/orld is 
developmentally structured 

Moving fix)m pre-trip assumptions and initial site dialogue to being able to explain discrete 
factual assertions entails botii an aesthetics of communication and objectification of discourse, 
(Geertz, 1973). In tiie course of relating cultural judgments and cultural dercriptions, tiie study 
tour member is confounded by his/her undefined role. Not quite a tourist and not quite "there on 
business", tiie tour member is particularly vukerable to discursive sabotage (Jules-Rosette, 1989) 
where the accuracy of cultural awareness relies upon tiie content gleaned from the participant's 
inquiries. The aestiietic aspect of tiiis process relates to tiie style in which information is 
communicated first by the native source to the study tour member and tiien by the tour 
member/teacher to a broader audience of students back home. 

Harvey Sacks (1972) outiines rales of economy, adequate reference, and consistency to 
explain this discursive process. Through an adequate reference, a term appears as an appropriate 
and recognizable member of a category. Under the consistency rule, the term must be used in an 
uniform manner to classify membei's of die same set or collection. Consider die Japanese 
category, "sensei", or the American category, "Southerners" as a collective identification. You 
can go on and inventory a number of cultural traits according to which tiiese people may be 
classified These traits are classed witii reference to livelihood, material culture, social 
arrangements, and political institutions. The danger of traits is tiiat tiiey can be easily translated 
into generalizations to support the tour member's decisions for assigning a particular identification 
to a category of people, objects or events. 

Understanding another requires reciprocity and playing common game rules. This game 
involves playing upon a set of mutual correlations, or semiotic oppositions, in a manner that 
hinders attempts to transform dialogue into factual data about lineages, kmship, and ' ^!lage social 
structure. Though both are labelled F/m tVlorW countries, tiie East-West world view continua 
makes it particularly difficult for short-term viators to transcend more tiian tiie superficial rules of 
each host country. An essential part of learning operational rules requires dialogical exchanges 
witii natives. Though it is dangerous to generalize, framing questions in Japan tended to be 
different than in tiie United States. For example, in Japan a direct question at an untimely moment 
can be very destructive, merely by virtue of its directness. Direct questions are to be avoided, 
reflecting a Japanese value of allowing tilings to remain implicit ratiier tiian calling attention to 
oneself, making oneself clear and taking responsibility for whatever consequences were incurred. 
If one asks "Who are you?" in order to obtain an explicit signifier indicating a person's identity as a 
member of a specific collection or subgroup, one often gets tiie answer "A Japanese". What 
appears to be a direct question often becomes an indirect speech act (i.e., those utterances that 
indicate the appearance of disturbing topics in discourse and therefore require circumlocution. 
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(Searle, 1975X In fact, the subject is omitted in about 60% of spoken Japanese sentences (Morley, 
1985), Gathering answers to personal questions is particularly diflRculL Many US visitors 
remaii«;;d about the reticence of their Japanese acquaintances to state what they actually knew- 

The tentative, self-effacing, amb^ous use of language is known as aimai and this may be 
attributed to sociological and psychological factors. For one, the historical impact of the autocratic 
Tokugawa shogunate (1603-1867) where those who practiced verbal discretion definitely had 
longer life spans. As the old Japanese saying goes, the nail that sticks out gets banged down, A 
coTft Japanese value appears to be harmony, 

which they seek to achieve by a subtle process of mutual understanding, almost intuition, 
rather than by a sharp analysis of conflicting views or by clear-cut decisions, whether made 
by one-man dictates or majority votes. Decisions, tiiey feel, should not be left up to any 
one man but should be arrived at by consultations and committee work. Consensus is the 
goal, (Reischauer, 1981,p.l35) 

The Japanese itineraries contained more instances of aimai than die American itineraries. American 

plans were frequently filled with details about desired interactive contacts with individuals whereas 

the Japanese emphasis seemed to concentrate more on visual experiences widi physical sites and 

organizations versus individuals, 

Morley's (1985) analysis of this phenomenon mentions Japanese proverbs testifying to the 
advantages of silence such as "a defeated warrior does not speak" or sannen kata-hOy meaning tiiat 
a warrior should not allow himself to show emotion more tiian once in three years. Much is 
suggested by indirection or vague implication. According to Reischauer (1981), the Japanese have 
a word for tiiis indirect and/or nonverbal meeting of die mind, haragei, whose literal translation is 
"die art of die belly". Apparently, Americans are still struggling witii aimai when attempting to 
interpret Japan's post World War n constitution, conceptualized under die Marshall Plan in English 
but open to different interpretations in the operational Japanese version. 

Anotiier explanation of aimai can be derived from a remark made by the psychologist 
Okonogi Keigo is a discussion of haji, die Japanese concept of shame, ''Haji is the characteristic 
feeling of people who are dependent upon on a group, taking the fomi of a vague fear of rumour, 
gossip, of possible ostracism. In die day to day life of pe^^ple who are sensitive to this kind of 
shame die essential thing is accordingly not to express oneself unequivocally," Even when 
translated into English, many American study tour members formed die conclusion tiiat die 
Japanese approached questions in a deliberate way not to be understood by outsiders. 

The foreigner or gaijin in Japan is treated very politely but always as an outsider-no matter 
how long he lives in Japan or how deeply he may be involved m its life. Extended interpersonal 
experiences were avoided by most Japanese that tour members met for die first time. This 
appeared to be in direct juxtaposition to American preferences. Long time resident and Japanophile 
Reischauer captured it well when he said: 
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If he (foreigner) speaks any Japanese at all, no niatter how badly, he is praised for 
this remaricable accomplishnoient, as though he were an idiot child who suddenly 
showed a streak of intelligence. If he is knowledgeable about Japan, he may be 
asked deferentially his opinions and told he knows more than they do themselves, 
but his opinions are regarded always as those of an outsider, not an insider 
(Reischauer, 1981, p 405) 

t the answer true or false, personal or classificatory when there *s a difference in motives 
between the questioner and the informant and both parties wish to transform their exchan^ into 
culturally pertinent material? Umberto Eco (1976, pp. 271-1) uses a game model to describe the 
process whereby participants in an aesthetic communication accept an underlying set of rules or 
"system of mutual correlations" and then break or stretch these rules to produce an unanticipated 
effect He goes on to state that "the aesthetic text is like a multiple match played by different teams 
at a time, each of whom follows (or breaks) die rules of tiieir own game" (p, 271), The result, he 
concludes is " a semiotic design which cunningly gives the impression of nonsemiosis. The 
aesthetic expression requires a special interpretative leap (p, 38), Botii American and Japanese 
travel itineraries scheduled experiences which seem to predestine, being captured in stereotypical 
tourist rituals of economic and recreational nature. 

Studying the educational system coupled witii site visits to schools and interaction witii 
children in each country proved to be an expeditious way of experiencing the game and identifying 
critical components of bcmiotic design. All groups scheduled school contacts, though this 
appeared to be addressed much more concretely by the American tours. Morley (1985) challenges 
us to observe simple school rituals which offer deep cultural insights given die centralized nature of 
the Japanese educational system. Consider how children are taught to write letters in Japan. He 
observed tiiat the nationalized Japanese curriculum informs children tiiat a letter should never begin 
witii oneself but witii an inquiry into the healtii and tiie general well-being of its recipient This 
should be followed by comments about the weather. For each month a set phrase was provided. 
Only tiien was it considered proper to broach the real topic of the letter- In our experience as 
evaluators working with Japanese, this phenomena also occurred witii phone an^' ax inquiries and 
responses. 

For the individual study tour number as well as evaluators, collecting descriptions and 
judgments of an cultural experience entails similar distinctive literary rules and figurative devices. 
Empirically dominated strategies such as die objectives based or decision making approaches seem 
to be disparaged as rikutsuppoi or "smacking too much of logic tiiat ignores reality". Coupled 
with a prevalent Weltanf>chauung of fatalism and passive expectancy, tiiis explains the frequent use 
of traditional and predictable events as an approach to cultural awareness by tiie Japanese 
itineraries. Perhaps tjus rikutsuppoi accounts in part for die absence of the concept " program 
evaluator" in Japanese. 
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Minami Hiioshi posits a national masochisnoL that perceives of acceptance of the negative as 
spiritual strength. One sees this in numerous Japanese aesthetic productions, notably the work of 
Yokio Mishima* Fusoku-shugU a fourteenth century theory, of Insufficiency which identified the 
most profound aestiietic experience as lying in that which had been withheld and left unsaid, 
appears very much in the fabric of riKxiem Japan and goes against the grain of Western evaluators* 

Consider the rules inherent in communicating with people who assume their language is a 
means of retention and exclusion (Japanese) versus one of release and expression (American)* For 
example, the courtesy language adopted in Japan for exchanges outside the uchi (a social unit 
related by blood and common locale, but also covering any pcwns brought together physically 
and sphitually in a common cause) is circumspect, distrustful, and avoids sensitive topics* 
American tour members consistentiy cited tiiis phenomenon. Self-sufficiency exists within one's 
uchi rather than within oneself* This courtesy language-both oral and written, verbal and 
non verbal-is indicated by self-effacement expressions, cautious references, superfluous 
agreement, solicitude and deference* Interestingly, the rules that controlled behavior in Japan 
where not always in force when Japanese travelled outside their own country* 

The research and evaluation acts of exposure, analysis and critique does not extend beyond 
one's uchi or organization without resistance* More than a continuous search to sell your skill to 
the highest bidder, a job in Japan means pride in being part of something big and significant for the 
rest of your working life* And this group identification is pervasive, extending fiiom schools, 
especially at the college level, to PTAs, hobby groups. Rotary, new religions, and sightseeing 
groups (Reischauer, 1981)* Given the American values of forthrightness, emotional 
demonstrations, conjfrontation, physical strength and individual independence, these Japanese 
language signs are rejected by US outsiders before they even have time to make them overt Those 
gaijin who spoke fluent Japanese went througl'f dissociation from their own culture, even a sort of 
hostility on the road to their fluency and recouated vales of how Japanese had refused to hear them 
speak Japanese and tried to answer in English rather than recognize a gaijiris ability to play the 
game* A solution for even fluent speakers is the wide use of go-betweens and established network 
of contacts. The go-between scouts out the views of both sides, finds ways around problems and 
if the emotions become too vivid, end the exchange without the danger of a direct (and attention 
drawing) confrontation* On the otherhand, Americans seemed all to eager (or relieved?) to 
discover Japanese typically spoke much better English and over time camo to expect this facility* 

What emerges is the critical importance of fluency in verbal and nonverbal language for 
both insiders and outsiders* Even if one masters verbal Japanese replete with its numerous 
incomplete sentences, there are so many ambiguous pre-speech mood orienting sounds required 
for one's own productions or encouraging others expression (aizuchi ), but little spontaneous and 
explicit body language which seems to be the reverse of Americans. The closest analogy we can 
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think of in the American scene is the intenogatory dissembling of the TV detective, Columbo. 
Accompanying "appropriate" verbal and nonverbal exchanges are its accoutrements such as context 
appropriate dress or uniforms in keeping with the occasion (basho-gara wakir/iaezu ) and the 
exchange of meishi (business cards ) at even the most apparendy casual encounter. Meishi helped 
fill the semiotic gaps by categorizing the individual by name, organization, rank and function. In 
the US we have no such mechanism, the absence of which suggests another cultural sign. 

Major learnings and emerging polemics 
Evaluation is ever present and will continue to operate, both overtly and covertly. The 
evaluation is only as good as the shared Umwelt that the evaluator is able to coalesce and make 
overt among all significant players. Approaching this 'willed' reality or new Umwelt can be 
facilitated by Polanyi's (1962, 1966, 1975, 1978) constructs of collecting multiple perceptions in 
order to approach a shared reality, trying to make tacit perceptions overt rather than being 
unconsciously controlled by these perceptions, and extended dwelling in an experience to 
experience more than the surface phenomena* In essence, out of these diverse perceptions we can 
dentifyi critical polemics as signposts which expand but do not fix our capability to describe and 
judge. By presenting thirteen polemics emerging firom our bi-national evaluation experiences as 
signs, we hope to stimulate discussion of how semiotics informs evaluation inquiry. Table 2 
summariizes these polemics. 

insert Table 2 about here 

How can balanced efforts to "improve" educational programs be achieved in changing 
political-economic climates v/hen these activities are supported by diverse intra-national and 
international purposes? One of the major challenges of this evaluation quickly became the 
management of nine geographically diverse sets of quantitative and qualitative data in the United 
States and two in Japan. How does an external evaluation communicate this morass of local, 
re^onal and national programming to a bi-national board of trustees to facilitate unified futures 
planning-in essoice, to negotiate a shared Umwelt? The use of polemics proved very helpful 
here. 

Woricable notions of "core cultural curricula" and "global education" have been only 
partially realized world-wide. Thft road to global concepts such as bi-culturalism within curricular 
contexts is sequential. Initial emphasis must be given to localization. Ultimately, this examination 
has been targeted on the affordability of empiricism and "proof* versus "improvement" paradigms 
in more localized and regional settings. It appears that the program leadership must constandy 
balance an egalitarian versus an elitist polemic. In the extent of our evaluation experiences, 
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enhanced cultural awareness and curriculum infusion must replace established systems of material 
and symbolic exchanges between political and educational elites, communities, parents and direct 
service providers-the teachers. The more the country is in flux economically, the more the 
question of value imposition resulting ftom reform nK)vements looms. It is definitely a challenge 
for the foundation to be both eclectic and capable of listening to diverse values, ecolo^cal in light 
of limited resources and personnel and identifying common bi-national values so that a synthetic 
base of "program improvement" can be perceived as helpful. 

What kind of evaluation paradigm (car myth according to semioticians) is valuable? Value 
must be considered in some kind of culturally sensitive semiotic context, making cross-national 
applications difficult to achieve. For transferability of evaluation, trust must be pres-^.nt This is a 
prospect difficult to achieve in one's own culture, let alone attempting to cross East- West 
boundaries. Much of die actual use of evaluation in the US would seem to have limited 
applicability in Japan tiiough perception-based nxxiels would appear to be the most consonant 
"Levels of consciousness" and ethnocentricity appear to influence significantly die concepts, 
applications, timeline, and ultimate usefulness of evaluation. Even in die US, the struggle to orient 
program evaluation to an improvement-directed stance is constant; many evaluation audiences have 
strongly reinforced perceptions of evaluations being exclusively directed toward "proof* of 
programs. Teaching die evaluated diat perceptions determine their reality is a challenge that must 
be confirmed early (as well as determining what weight will be accorded to representative 
perceptions of all the major stakeholders). 

Who verifies die procedures of translation and evaluation? And how does the 
evaluator discern accurate information fix)m misinformation? Avoiding manipulation and achieving 
holism demands risk-taking on die part of the evaluated and die evaluators aUke. Given that we are 
dealing widi two cultures widi distinctly different core value systems, die process of paradigm 
creation cannot rely solely in die hands of one evaluator. Multiple perceptions of diverse audiences 
in both cultures must be obtained in order to more accurately regulate the interaction of 
assumptions and representations for recognition. If we assume Benzon & Hays* (1987) 
contention diat physiognomic and visually holistic representations tend to be die preferred mode in 
Eastern cultures and Western perceptions tend to display propositional and verbal representations, 
360 degrees of perceptual descriptions must be collected in an effort to make die implicit explicit 
In fact, each distinct constituency represented in the evaluation risks losing some of their unique 
commcKiity in die process of negotiating a greater shared reality. 

The discrepancies between pre-and post-trip perceptions leads you to consciously seek die 
mydi or paradigm which is built upon verbal and iconic parts. You then search for the part where 
the discrepancies are greatest, that will tell you where your awareness has changed. But what if 
you are like Arm Tylef s Accidental Tourist, exposed to other cultures but not seeking 
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discrepancies (in fact, assiduously avoiding them) because you have been captured in your own 
cultural cacoon? Or what happens when the traveller and the native informant power consciously 
hold contrary positions? Potentially, what is at risk is the transformation of the knowledge base 
of the traveller versus the transformation of the native, A discrepancy analysis remains incomplete 
without negotiation of perceptions of diverse audiences and stakeholders. Instruction in cultural 
metaphors can lead to recognition and over time to cultural awareness and fluent reading of signs. 
If we merely consider the dual meaning of the concept "alien", we may conclude that practice in 
reading signs in other cultures helps develop a conscious reading in one's ov/n culture. 

Summary 

Even within educational program evaluation, the major paradigms fell short of a theory that 
meets the semiotic criteria set forth in tiiis paper. When one examines the works of evaluators such 
as Stuffelbeam one jfinds the necessary components of a paradigm. While their models are 
mechanically total, they exclude the negotiation of values when major questions of focus are being 
determined as well as fail to predefine or make overt the values to be applied in making judgments, 
Michael Scriven, a philosopher as well as an evaluation theorist, deals with values but his external 
approach assumes an elitist orientation, Scrivtti suggests that values must be brought into the 
schema from outside sources for judgments to be made, but does not set up the necessary 
dynamics of negotiating major questions to focus the study, Robert Stake is concerned about 
judgments and values and does include the "outsider" evaluatofs relative and absolute judgements 
while avoiding the pitfalls of an absolutist position regarding a program'^ goals, 

A semiotic paradigm is clarified by the conscious inch^sion of p( icmics in order to explicate 
program myths. In essence, we propose approaching evaluation as both a native and an alieriy an 
insider and a disciplined outsider. What is needed is a paradigm which negotiates major questions 
openly with all participants; gatiiers quantitative and qualitative sources of evidence; and applies 
standards or value criteria which are neither elitist ftom the pomt of view of the evaluator, nor 
captured by the tacit agendas of the stakeholders. 
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Table 1. The Perception-Based Evaluation: a Semiotic Paradigm 



Boxl 



Box 2 



Box 3 



What major questions should 
be asked in the evaluation? 



What sources of 
evidence can be found? 



What quality/value 
components will be 
applied? 



Questions having a holistic 
emphasis wherein all 
syntheses are repeatedly 
tested against experiential 
perceptions 

Questions which are helpful 
through a dynamic dialogue, 
negotiation, and pedagogical 
action or make visible the 
values being synthesized by 
both evaluator and evaluatee 

Questions which expand the 
range of major questions 
asked by recognizing 
distortions of positivist and 
reductionist biases 

Questions which emphasize 
the vulnerability of the 
evaluator to evaluate in an 
effort to avoid oppression of 
evaluatee 

Questions which facilitate die 
planning of program futures 



Multiple methodologies are 
used in an effort to survey 
the "whole" of reality: e^g,, 
descriptions, observations, 
subjective impressions, 
scientific metiiod, interviews, 
perceptions, & artifacts 

Should be those sources 
which are closest in 
agreement with multiple 
perceivers 

Both "hard" and V^ft^'data 
are systematically used to 
obtain an accurate picture 
of reality 



Judgments and values 
grounded in perceptions 
and are overt 

Openness to seeking multiple 
visions of reality and the 
totality of experience 

Orientation to the future, 
extendability, and transfer- 
ability of levels of 
consciousness 

Discussion, mutual respon- 
sibility, mutual criticism, 
and negotiation 

Respi^ct for core values 
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Table li Polemics of an Evalual 


tion Semiotic 
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